Attrition in Conversational Agent-Delivered Mental Health Interventions: Systematic Review and Meta-Analysis.

BACKGROUND: Conversational agents (CAs) or chatbots are computer programs that mimic human conversation. They have the potential to improve access to mental health interventions through automated, scalable, and personalized delivery of psychotherapeutic content. However, digital health interventions, including those delivered by CAs, often have high attrition rates. Identifying the factors associated with attrition is critical to improving future clinical trials. OBJECTIVE: This review aims to estimate the overall and differential rates of attrition in CA-delivered mental health interventions (CA interventions), evaluate the impact of study design and intervention-related aspects on attrition, and describe study design features aimed at reducing or mitigating study attrition. METHODS: We searched PubMed, Embase (Ovid), PsycINFO (Ovid), Cochrane Central Register of Controlled Trials, and Web of Science, and conducted a gray literature search on Google Scholar in June 2022. We included randomized controlled trials that compared CA interventions against control groups and excluded studies that lasted for 1 session only and used Wizard of Oz interventions. We also assessed the risk of bias in the included studies using the Cochrane Risk of Bias Tool 2.0. Random-effects proportional meta-analysis was applied to calculate the pooled dropout rates in the intervention groups. Random-effects meta-analysis was used to compare the attrition rate in the intervention groups with that in the control groups. We used a narrative review to summarize the findings. RESULTS: The systematic search retrieved 4566 records from peer-reviewed databases and citation searches, of which 41 (0.90%) randomized controlled trials met the inclusion criteria. The meta-analytic overall attrition rate in the intervention group was 21.84% (95% CI 16.74%-27.36%; I2=94%). Short-term studies that lasted ≤8 weeks showed a lower attrition rate (18.05%, 95% CI 9.91%- 27.76%; I2=94.6%) than long-term studies that lasted >8 weeks (26.59%, 95% CI 20.09%-33.63%; I2=93.89%). Intervention group participants were more likely to attrit than control group participants for short-term (log odds ratio 1.22, 95% CI 0.99-1.50; I2=21.89%) and long-term studies (log odds ratio 1.33, 95% CI 1.08-1.65; I2=49.43%). Intervention-related characteristics associated with higher attrition include stand-alone CA interventions without human support, not having a symptom tracker feature, no visual representation of the CA, and comparing CA interventions with waitlist controls. No participant-level factor reliably predicted attrition. CONCLUSIONS: Our results indicated that approximately one-fifth of the participants will drop out from CA interventions in short-term studies. High heterogeneities made it difficult to generalize the findings. Our results suggested that future CA interventions should adopt a blended design with human support, use symptom tracking, compare CA intervention groups against active controls rather than waitlist controls, and include a visual representation of the CA to reduce the attrition rate. TRIAL REGISTRATION: PROSPERO International Prospective Register of Systematic Reviews CRD42022341415; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42022341415.

Conversational Agents in Health Care: Expert Interviews to Inform the Definition, Classification, and Conceptual Framework.

BACKGROUND: Conversational agents (CAs), or chatbots, are computer programs that simulate conversations with humans. The use of CAs in health care settings is recent and rapidly increasing, which often translates to poor reporting of the CA development and evaluation processes and unreliable research findings. We developed and published a conceptual framework, designing, developing, evaluating, and implementing a smartphone-delivered, rule-based conversational agent (DISCOVER), consisting of 3 iterative stages of CA design, development, and evaluation and implementation, complemented by 2 cross-cutting themes (user-centered design and data privacy and security). OBJECTIVE: This study aims to perform in-depth, semistructured interviews with multidisciplinary experts in health care CAs to share their views on the definition and classification of health care CAs and evaluate and validate the DISCOVER conceptual framework. METHODS: We conducted one-on-one semistructured interviews via Zoom (Zoom Video Communications) with 12 multidisciplinary CA experts using an interview guide based on our framework. The interviews were audio recorded, transcribed by the research team, and analyzed using thematic analysis. RESULTS: Following participants' input, we defined CAs as digital interfaces that use natural language to engage in a synchronous dialogue using ≥1 communication modality, such as text, voice, images, or video. CAs were classified by 13 categories: response generation method, input and output modalities, CA purpose, deployment platform, CA development modality, appearance, length of interaction, type of CA-user interaction, dialogue initiation, communication style, CA personality, human support, and type of health care intervention. Experts considered that the conceptual framework could be adapted for artificial intelligence-based CAs. However, despite recent advances in artificial intelligence, including large language models, the technology is not able to ensure safety and reliability in health care settings. Finally, aligned with participants' feedback, we present an updated iteration of the conceptual framework for health care conversational agents (CHAT) with key considerations for CA design, development, and evaluation and implementation, complemented by 3 cross-cutting themes: ethics, user involvement, and data privacy and security. CONCLUSIONS: We present an expanded, validated CHAT and aim at guiding researchers from a variety of backgrounds and with different levels of expertise in the design, development, and evaluation and implementation of rule-based CAs in health care settings.

Scope, Characteristics, Behavior Change Techniques, and Quality of Conversational Agents for Mental Health and Well-Being: Systematic Assessment of Apps.

BACKGROUND: Mental disorders cause substantial health-related burden worldwide. Mobile health interventions are increasingly being used to promote mental health and well-being, as they could improve access to treatment and reduce associated costs. Behavior change is an important feature of interventions aimed at improving mental health and well-being. There is a need to discern the active components that can promote behavior change in such interventions and ultimately improve users' mental health. OBJECTIVE: This study systematically identified mental health conversational agents (CAs) currently available in app stores and assessed the behavior change techniques (BCTs) used. We further described their main features, technical aspects, and quality in terms of engagement, functionality, esthetics, and information using the Mobile Application Rating Scale. METHODS: The search, selection, and assessment of apps were adapted from a systematic review methodology and included a search, 2 rounds of selection, and an evaluation following predefined criteria. We conducted a systematic app search of Apple's App Store and Google Play using 42matters. Apps with CAs in English that uploaded or updated from January 2020 and provided interventions aimed at improving mental health and well-being and the assessment or management of mental disorders were tested by at least 2 reviewers. The BCT taxonomy v1, a comprehensive list of 93 BCTs, was used to identify the specific behavior change components in CAs. RESULTS: We found 18 app-based mental health CAs. Most CAs had <1000 user ratings on both app stores (12/18, 67%) and targeted several conditions such as stress, anxiety, and depression (13/18, 72%). All CAs addressed >1 mental disorder. Most CAs (14/18, 78%) used cognitive behavioral therapy (CBT). Half (9/18, 50%) of the CAs identified were rule based (ie, only offered predetermined answers) and the other half (9/18, 50%) were artificial intelligence enhanced (ie, included open-ended questions). CAs used 48 different BCTs and included on average 15 (SD 8.77; range 4-30) BCTs. The most common BCTs were 3.3 "Social support (emotional)," 4.1 "Instructions for how to perform a behavior," 11.2 "Reduce negative emotions," and 6.1 "Demonstration of the behavior." One-third (5/14, 36%) of the CAs claiming to be CBT based did not include core CBT concepts. CONCLUSIONS: Mental health CAs mostly targeted various mental health issues such as stress, anxiety, and depression, reflecting a broad intervention focus. The most common BCTs identified serve to promote the self-management of mental disorders with few therapeutic elements. CA developers should consider the quality of information, user confidentiality, access, and emergency management when designing mental health CAs. Future research should assess the role of artificial intelligence in promoting behavior change within CAs and determine the choice of BCTs in evidence-based psychotherapies to enable systematic, consistent, and transparent development and evaluation of effective digital mental health interventions.

Evaluating TESLA-G, a gamified, telegram-delivered, quizzing platform for surgical education in medical students: protocol for a pilot randomised controlled trial.

INTRODUCTION: Online multiple-choice question (MCQ) quizzes are popular in medical education due to their ease of access and ability for test-enhanced learning. However, a general lack of motivation among students often results in decreasing usage over time. We aim to address this limitation by developing Telegram Education for Surgical Learning and Application Gamified (TESLA-G), an online platform for surgical education that incorporates game elements into conventional MCQ quizzes. METHODS AND ANALYSIS: This online, pilot randomised control trial will be conducted over 2 weeks. Fifty full-time undergraduate medical students from a medical school in Singapore will be recruited and randomised into an intervention group (TESLA-G) and an active control group (non-gamified quizzing platform) with a 1:1 allocation ratio, stratified by year of study.We will evaluate TESLA-G in the area of endocrine surgery education. Our platform is designed based on Bloom's taxonomy of learning domains: questions are created in blocks of five questions per endocrine surgery topic, with each question corresponding to one level on Bloom's taxonomy. This structure promotes mastery while boosting student engagement and motivation. All questions are created by two board-certified general surgeons and one endocrinologist, and validated by the research team. The feasibility of this pilot study will be determined quantitatively by participant enrolment, participant retention and degree of completion of the quizzes. The acceptability of the intervention will be assessed quantitatively by a postintervention learner satisfaction survey consisting of a system satisfaction questionnaire and a content satisfaction questionnaire. The improvement of surgical knowledge will be assessed by comparing the scores of preintervention and postintervention knowledge tests, which consist of separately created questions on endocrine surgery. Retention of surgical knowledge will be measured using a follow-up knowledge test administered 2 weeks postintervention. Finally, qualitative feedback from participants regarding their experience will be obtained and thematically analysed. ETHICS AND DISSEMINATION: This research is approved by Singapore Nanyang Technological University (NTU) Institutional Review Boards (Reference Number: IRB-2021-732). All participants will be expected to read and sign a letter of informed consent before they are considered as recruited into the study. This study poses minimal risk to participants. Study results will be published in peer-reviewed open-access journals and presented in conference presentations. TRIAL REGISTRATION NUMBER: NCT05520671.

Development of "LvL UP 1.0": a smartphone-based, conversational agent-delivered holistic lifestyle intervention for the prevention of non-communicable diseases and common mental disorders.

Background: Non-communicable diseases (NCDs) and common mental disorders (CMDs) are the leading causes of death and disability worldwide. Lifestyle interventions via mobile apps and conversational agents present themselves as low-cost, scalable solutions to prevent these conditions. This paper describes the rationale for, and development of, "LvL UP 1.0″, a smartphone-based lifestyle intervention aimed at preventing NCDs and CMDs. Materials and Methods: A multidisciplinary team led the intervention design process of LvL UP 1.0, involving four phases: (i) preliminary research (stakeholder consultations, systematic market reviews), (ii) selecting intervention components and developing the conceptual model, (iii) whiteboarding and prototype design, and (iv) testing and refinement. The Multiphase Optimization Strategy and the UK Medical Research Council framework for developing and evaluating complex interventions were used to guide the intervention development. Results: Preliminary research highlighted the importance of targeting holistic wellbeing (i.e., both physical and mental health). Accordingly, the first version of LvL UP features a scalable, smartphone-based, and conversational agent-delivered holistic lifestyle intervention built around three pillars: Move More (physical activity), Eat Well (nutrition and healthy eating), and Stress Less (emotional regulation and wellbeing). Intervention components include health literacy and psychoeducational coaching sessions, daily "Life Hacks" (healthy activity suggestions), breathing exercises, and journaling. In addition to the intervention components, formative research also stressed the need to introduce engagement-specific components to maximise uptake and long-term use. LvL UP includes a motivational interviewing and storytelling approach to deliver the coaching sessions, as well as progress feedback and gamification. Offline materials are also offered to allow users access to essential intervention content without needing a mobile device. Conclusions: The development process of LvL UP 1.0 led to an evidence-based and user-informed smartphone-based intervention aimed at preventing NCDs and CMDs. LvL UP is designed to be a scalable, engaging, prevention-oriented, holistic intervention for adults at risk of NCDs and CMDs. A feasibility study, and subsequent optimisation and randomised-controlled trials are planned to further refine the intervention and establish effectiveness. The development process described here may prove helpful to other intervention developers.

Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments.

BACKGROUND: Rapid proliferation of mental health interventions delivered through conversational agents (CAs) calls for high-quality evidence to support their implementation and adoption. Selecting appropriate outcomes, instruments for measuring outcomes, and assessment methods are crucial for ensuring that interventions are evaluated effectively and with a high level of quality. OBJECTIVE: We aimed to identify the types of outcomes, outcome measurement instruments, and assessment methods used to assess the clinical, user experience, and technical outcomes in studies that evaluated the effectiveness of CA interventions for mental health. METHODS: We undertook a scoping review of the relevant literature to review the types of outcomes, outcome measurement instruments, and assessment methods in studies that evaluated the effectiveness of CA interventions for mental health. We performed a comprehensive search of electronic databases, including PubMed, Cochrane Central Register of Controlled Trials, Embase (Ovid), PsychINFO, and Web of Science, as well as Google Scholar and Google. We included experimental studies evaluating CA mental health interventions. The screening and data extraction were performed independently by 2 review authors in parallel. Descriptive and thematic analyses of the findings were performed. RESULTS: We included 32 studies that targeted the promotion of mental well-being (17/32, 53%) and the treatment and monitoring of mental health symptoms (21/32, 66%). The studies reported 203 outcome measurement instruments used to measure clinical outcomes (123/203, 60.6%), user experience outcomes (75/203, 36.9%), technical outcomes (2/203, 1.0%), and other outcomes (3/203, 1.5%). Most of the outcome measurement instruments were used in only 1 study (150/203, 73.9%) and were self-reported questionnaires (170/203, 83.7%), and most were delivered electronically via survey platforms (61/203, 30.0%). No validity evidence was cited for more than half of the outcome measurement instruments (107/203, 52.7%), which were largely created or adapted for the study in which they were used (95/107, 88.8%). CONCLUSIONS: The diversity of outcomes and the choice of outcome measurement instruments employed in studies on CAs for mental health point to the need for an established minimum core outcome set and greater use of validated instruments. Future studies should also capitalize on the affordances made available by CAs and smartphones to streamline the evaluation and reduce participants' input burden inherent to self-reporting.

Smartphone and Mobile App Use Among Physicians in Clinical Practice: Scoping Review.

BACKGROUND: Health care professionals are increasingly using smartphones in clinical care. Smartphone use can affect patient quality of care and clinical outcomes. OBJECTIVE: This scoping review aimed to describe how physicians use smartphones and mobile apps in clinical settings. METHODS: We conducted a scoping review using the Joanna Briggs Institute methodology and reported the results according to PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines. We used the following databases in our literature search: MEDLINE, Embase, Cochrane Library, Web of Science, Google Scholar, and gray literature for studies published since 2010. An additional search was also performed by scanning the reference lists of included studies. A narrative synthesis approach was used. RESULTS: A total of 10 studies, published between 2016 and 2021, were included in this review. Of these studies, 8 used surveys and 2 used surveys with focus group study designs to explore smartphone use, its adoption, experience of using it, and views on the use of smartphones among physicians. There were studies with only general practitioners (n=3), studies with only specialists (n=3), and studies with both general practitioners and specialists (n=4). Physicians use smartphones and mobile apps for communication (n=9), clinical decision-making (n=7), drug compendium (n=7), medical education and training (n=7), maintaining health records (n=4), managing time (n=4), and monitoring patients (n=2) in clinical practice. The Medscape medical app was frequently used for information gathering. WhatsApp, a nonmedical app, was commonly used for physician-patient communication. The commonly reported barriers were lack of regulatory oversight, privacy concerns, and limited Wi-Fi or internet access. The commonly reported facilitator was convenience and having access to evidence-based medicine, clinical decision-making support, and a wide array of apps. CONCLUSIONS: Smartphones and mobile apps were used for communication, medical education and training, clinical decision-making, and drug compendia in most studies. Although the benefits of smartphones and mobile apps for physicians at work were promising, there were concerns about patient privacy and confidentiality. Legislation is urgently needed to protect the liability of health care professionals using smartphones.

Smartphone apps for point-of-care information summaries: systematic assessment of the quality and content.

BACKGROUND: Clinicians need easy access to evidence-based information to inform their clinical practice. Point-of-care information summaries are increasingly available in the form of smartphone apps. However, the quality of information from the apps is questionable as there is currently no regulation on the content of the medical apps. OBJECTIVES: This study aimed to systematically assess the quality and content of the medical apps providing point-of-care information summaries that were available in two major app stores. We evaluated apps designed specifically for healthcare professionals and assessed their content development, editorial policy, coverage of medical conditions and trustworthiness. METHODS: We conducted a systematic assessment of medical apps providing point-of-care information summaries available in Google Play and Apple app stores. Apps launched or updated since January 2020 were identified through a systematic search using 42matters. Apps meeting the inclusion criteria were downloaded and assessed. The data extraction and app assessment were done in parallel and independently by at least two reviewers. Apps were evaluated against the adapted criteria: (1) general characteristics, (2) content presentation of the summaries, (3) editorial quality, (4) evidence-based methodology, (5) coverage (volume) of the medical conditions, (6) usability of apps and (7) trustworthiness of the app based on HONcode principles. HONcode principles are guidelines used to inform users about the credibility and reliability of health information online. The results were reported as a narrative review. RESULTS: Eight medical apps met the inclusion criteria and were systematically appraised. Based on our evaluation criteria, UpToDate supported 16 languages, and all other apps were English. Bullet points and brief paragraphs were used in all apps, and only DynaMed and Micromedex and Pathway-medical knowledge provided a formal grading system for the strength of recommendations for all the medical conditions in their apps. All the other apps either lacked a formal grading system altogether or offered one for some of the medical conditions. About 30% of the editorial quality assessment and 47.5% of the evidence-based methodology assessment were unclear or missing. UpToDate contained the most point-of-care evidence-based documents with >10 500 documents. All apps except 5-Minute Clinical Consult and DynaMed and Micromedex were available for offline access. Only Medscape complied with the HONcode principles. CONCLUSIONS: Future apps should report a more detailed evidence-based methodology, be accessible for offline use and support search in more than one language. There should be clearer information provided in future apps regarding the declaration of authorship and conflict of interest.

Exploring the Perceptions of mHealth Interventions for the Prevention of Common Mental Disorders in University Students in Singapore: Qualitative Study.

BACKGROUND: Mental health interventions delivered through mobile health (mHealth) technologies can increase the access to mental health services, especially among university students. The development of mHealth intervention is complex and needs to be context sensitive. There is currently limited evidence on the perceptions, needs, and barriers related to these interventions in the Southeast Asian context. OBJECTIVE: This qualitative study aimed to explore the perception of university students and mental health supporters in Singapore about mental health services, campaigns, and mHealth interventions with a focus on conversational agent interventions for the prevention of common mental disorders such as anxiety and depression. METHODS: We conducted 6 web-based focus group discussions with 30 university students and one-to-one web-based interviews with 11 mental health supporters consisting of faculty members tasked with student pastoral care, a mental health first aider, counselors, psychologists, a clinical psychologist, and a psychiatrist. The qualitative analysis followed a reflexive thematic analysis framework. RESULTS: The following 6 main themes were identified: a healthy lifestyle as students, access to mental health services, the role of mental health promotion campaigns, preferred mHealth engagement features, factors that influence the adoption of mHealth interventions, and cultural relevance of mHealth interventions. The interpretation of our findings shows that students were reluctant to use mental health services because of the fear of stigma and a possible lack of confidentiality. CONCLUSIONS: Study participants viewed mHealth interventions for mental health as part of a blended intervention. They also felt that future mental health mHealth interventions should be more personalized and capable of managing adverse events such as suicidal ideation.

Subjective cognitive complaints in end-stage renal disease: a systematic review and meta-analysis.

Cognitive impairment is common in patients with end-stage renal disease (ESRD) and is associated with compromised quality of life and functional capacity, as well as worse clinical outcomes. Most previous research and reviews in this area were focused on objective cognitive impairment, whereas patients' subjective cognitive complaints (SCCs) have been less well-understood. This systematic review aimed to provide a broad overview of what is known about SCCs in adult ESRD patients. Electronic databases were searched from inception to January 2022, which identified 221 relevant studies. SCCs appear to be highly prevalent in dialysis patients and less so in those who received kidney transplantation. A random-effects meta-analysis also shows that haemodialysis patients reported significantly more SCCs than peritoneal dialysis patients (standardised mean difference -0.20, 95% confidence interval -0.38 to -0.03). Synthesis of longitudinal studies suggests that SCCs remain stable on maintenance dialysis treatment but may reduce upon receipt of kidney transplant. Furthermore, SCCs in ESRD patients have been consistently associated with hospitalisation, depression, anxiety, fatigue, and poorer quality of life. There is limited data supporting a strong relation between objective and subjective cognition but preliminary evidence suggests that this association may be domain-specific. Methodological limitations and future research directions are discussed.

What Is the Utility of Posters? Qualitative Study of Participants at a Regional Primary Healthcare Conference in Asia.

Background: Posters are used extensively as a mode of presentation at scientific conferences, but little is documented about their value to presenters or viewers. The study aimed to explore conference delegates' views and experiences of poster presentations, and their perceptions of the strengths and weaknesses of posters compared with oral presentations, and also to identify ways to enhance the educational value of posters. Method: This was a qualitative study using brief, semi-structured, face-to-face interviews amongst delegates at a 3-day Asia Pacific regional academic primary care conference in Singapore. Interviews were digitally recorded, transcribed verbatim, and their contents analysed thematically. Results: Eighty-nine interviews were analysed. Respondents were mainly early career researchers (58%), and a third were presenting (poster or oral) at the conference. Many positive attributes of posters were identified. For the viewers, these included the ability to gain a rapid overview of research activity (for "benchmarking", "updating", and "inspiration"); the ability to choose who to engage with and when, in contrast to the tightly scheduled oral sessions; and opportunity to discuss content in a leisurely and detailed fashion with the presenter. Presenters considered posters "less threatening" than oral presentations and valued posters for the networking opportunities they created. However, posters were reported to be more demanding on the skills of précis and their preparation was considered arduous and more expensive than an oral presentation. Posters were also perceived to have lower academic status and dominate the presenter's time at the conference, reducing the opportunities for them to see the work of others. Suggestions for incorporating technologies to enhance the impact of posters included QR codes to access more detailed information, pre-recorded presentations, and online interactive clarification sessions with poster authors. Conclusion: Posters are perceived as a valuable mode of presentation at scientific conferences by presenters and viewers. Their unique strengths challenge the perception that posters are somehow inferior to oral presentations, suggesting a need for their advantages to be promoted by researchers and conference organisers. The incorporation of technology within the traditional display may enhance poster utility. Given the time and money spent on academic conferences there is an urgent need to evaluate the different styles of presentation used at conferences and how they differ in their ability to impact on medical science knowledge and evidence-based clinical practice. Supplementary Information: The online version contains supplementary material available at 10.1007/s40670-022-01657-z.

Conversational Agents in Health Care: Scoping Review of Their Behavior Change Techniques and Underpinning Theory.

BACKGROUND: Conversational agents (CAs) are increasingly used in health care to deliver behavior change interventions. Their evaluation often includes categorizing the behavior change techniques (BCTs) using a classification system of which the BCT Taxonomy v1 (BCTTv1) is one of the most common. Previous studies have presented descriptive summaries of behavior change interventions delivered by CAs, but no in-depth study reporting the use of BCTs in these interventions has been published to date. OBJECTIVE: This review aims to describe behavior change interventions delivered by CAs and to identify the BCTs and theories guiding their design. METHODS: We searched PubMed, Embase, Cochrane's Central Register of Controlled Trials, and the first 10 pages of Google and Google Scholar in April 2021. We included primary, experimental studies evaluating a behavior change intervention delivered by a CA. BCTs coding followed the BCTTv1. Two independent reviewers selected the studies and extracted the data. Descriptive analysis and frequent itemset mining to identify BCT clusters were performed. RESULTS: We included 47 studies reporting on mental health (n=19, 40%), chronic disorders (n=14, 30%), and lifestyle change (n=14, 30%) interventions. There were 20/47 embodied CAs (43%) and 27/47 CAs (57%) represented a female character. Most CAs were rule based (34/47, 72%). Experimental interventions included 63 BCTs, (mean 9 BCTs; range 2-21 BCTs), while comparisons included 32 BCTs (mean 2 BCTs; range 2-17 BCTs). Most interventions included BCTs 4.1 "Instruction on how to perform a behavior" (34/47, 72%), 3.3 "Social support" (emotional; 27/47, 57%), and 1.2 "Problem solving" (24/47, 51%). A total of 12/47 studies (26%) were informed by a behavior change theory, mainly the Transtheoretical Model and the Social Cognitive Theory. Studies using the same behavior change theory included different BCTs. CONCLUSIONS: There is a need for the more explicit use of behavior change theories and improved reporting of BCTs in CA interventions to enhance the analysis of intervention effectiveness and improve the reproducibility of research.

Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER): Development of a Conceptual Framework.

BACKGROUND: Conversational agents (CAs), also known as chatbots, are computer programs that simulate human conversations by using predetermined rule-based responses or artificial intelligence algorithms. They are increasingly used in health care, particularly via smartphones. There is, at present, no conceptual framework guiding the development of smartphone-based, rule-based CAs in health care. To fill this gap, we propose structured and tailored guidance for their design, development, evaluation, and implementation. OBJECTIVE: The aim of this study was to develop a conceptual framework for the design, evaluation, and implementation of smartphone-delivered, rule-based, goal-oriented, and text-based CAs for health care. METHODS: We followed the approach by Jabareen, which was based on the grounded theory method, to develop this conceptual framework. We performed 2 literature reviews focusing on health care CAs and conceptual frameworks for the development of mobile health interventions. We identified, named, categorized, integrated, and synthesized the information retrieved from the literature reviews to develop the conceptual framework. We then applied this framework by developing a CA and testing it in a feasibility study. RESULTS: The Designing, Developing, Evaluating, and Implementing a Smartphone-Delivered, Rule-Based Conversational Agent (DISCOVER) conceptual framework includes 8 iterative steps grouped into 3 stages, as follows: design, comprising defining the goal, creating an identity, assembling the team, and selecting the delivery interface; development, including developing the content and building the conversation flow; and the evaluation and implementation of the CA. They were complemented by 2 cross-cutting considerations-user-centered design and privacy and security-that were relevant at all stages. This conceptual framework was successfully applied in the development of a CA to support lifestyle changes and prevent type 2 diabetes. CONCLUSIONS: Drawing on published evidence, the DISCOVER conceptual framework provides a step-by-step guide for developing rule-based, smartphone-delivered CAs. Further evaluation of this framework in diverse health care areas and settings and for a variety of users is needed to demonstrate its validity. Future research should aim to explore the use of CAs to deliver health care interventions, including behavior change and potential privacy and safety concerns.

Information needs and sources of information among people with depression and anxiety: a scoping review.

BACKGROUND: Previous studies have identified substantial unmet information needs in people with depression and anxiety. Sufficient information about the disorder, treatment, available services, and strategies for self-management is essential as it may influence quality of care and patients' quality of life. This scoping review aimed to provide a broad overview of information needs of people with depression and anxiety as well as the sources that they use to seek this information. METHODS: We included all primary research published in English that investigated information needs or information sources in people with depression or anxiety, with no restrictions imposed on the study design, location, setting, or participant characteristics. Six electronic databases (MEDLINE, Embase, PsycINFO, CINAHL, LISTA, Web of Science) and the grey literature (Google and Google Scholar) were searched for relevant studies published up to November 2021. Two reviewers independently screened articles and extracted data. Narrative synthesis was performed to identify key themes of information needs and information sources. Factors associated with information needs/sources such as demographic variables and symptom severity were also identified. RESULTS: Fifty-six studies (comprising 8320 participants) were included. Information needs were categorised into seven themes, including general facts, treatment, lived experience, healthcare services, coping, financial/legal, and other information. The most frequently reported needs in both people with depression and anxiety were general facts and treatment information. Subclinical samples who self-reported depressive/anxious symptoms appeared less interested in treatment information than patients with clinical diagnoses. Information sources were summarised into five categories: health professionals, written materials, media, interpersonal interactions, and organisational resources. Health professionals and media (including the internet) were the most frequently adopted and preferred sources. Although few studies have examined factors associated with information needs and information sources, there is preliminary evidence that symptom severity and disease subtypes are related to information needs/sources, whereas findings on demographic factors were mixed. CONCLUSIONS: Information needs appear to be high in people with depression and anxiety. Future research should examine differences between subgroups and associated factors such as the treatment course. Personalised information provision strategies are also needed to customise information according to individual needs and patient profiles. TRIAL REGISTRATION: The protocol of this scoping review was registered on Open Science Framework (OSF; link: https://doi.org/10.17605/OSF.IO/DF2M6 ).

Outcomes, Measurement Instruments, and Their Validity Evidence in Randomized Controlled Trials on Virtual, Augmented, and Mixed Reality in Undergraduate Medical Education: Systematic Mapping Review.

BACKGROUND: Extended reality, which encompasses virtual reality (VR), augmented reality (AR), and mixed reality (MR), is increasingly used in medical education. Studies assessing the effectiveness of these new educational modalities should measure relevant outcomes using outcome measurement tools with validity evidence. OBJECTIVE: Our aim is to determine the choice of outcomes, measurement instruments, and the use of measurement instruments with validity evidence in randomized controlled trials (RCTs) on the effectiveness of VR, AR, and MR in medical student education. METHODS: We conducted a systematic mapping review. We searched 7 major bibliographic databases from January 1990 to April 2020, and 2 reviewers screened the citations and extracted data independently from the included studies. We report our findings in line with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. RESULTS: Of the 126 retrieved RCTs, 115 (91.3%) were on VR and 11 (8.7%) were on AR. No RCT on MR in medical student education was found. Of the 115 studies on VR, 64 (55.6%) were on VR simulators, 30 (26.1%) on screen-based VR, 9 (7.8%) on VR patient simulations, and 12 (10.4%) on VR serious games. Most studies reported only a single outcome and immediate postintervention assessment data. Skills outcome was the most common outcome reported in studies on VR simulators (97%), VR patient simulations (100%), and AR (73%). Knowledge was the most common outcome reported in studies on screen-based VR (80%) and VR serious games (58%). Less common outcomes included participants' attitudes, satisfaction, cognitive or mental load, learning efficacy, engagement or self-efficacy beliefs, emotional state, competency developed, and patient outcomes. At least one form of validity evidence was found in approximately half of the studies on VR simulators (55%), VR patient simulations (56%), VR serious games (58%), and AR (55%) and in a quarter of the studies on screen-based VR (27%). Most studies used assessment methods that were implemented in a nondigital format, such as paper-based written exercises or in-person assessments where examiners observed performance (72%). CONCLUSIONS: RCTs on VR and AR in medical education report a restricted range of outcomes, mostly skills and knowledge. The studies largely report immediate postintervention outcome data and use assessment methods that are in a nondigital format. Future RCTs should include a broader set of outcomes, report on the validity evidence of the measurement instruments used, and explore the use of assessments that are implemented digitally.

Digital Education for Health Professionals: An Evidence Map, Conceptual Framework, and Research Agenda.

BACKGROUND: Health professions education has undergone major changes with the advent and adoption of digital technologies worldwide. OBJECTIVE: This study aims to map the existing evidence and identify gaps and research priorities to enable robust and relevant research in digital health professions education. METHODS: We searched for systematic reviews on the digital education of practicing and student health care professionals. We searched MEDLINE, Embase, Cochrane Library, Educational Research Information Center, CINAHL, and gray literature sources from January 2014 to July 2020. A total of 2 authors independently screened the studies, extracted the data, and synthesized the findings. We outlined the key characteristics of the included reviews, the quality of the evidence they synthesized, and recommendations for future research. We mapped the empirical findings and research recommendations against the newly developed conceptual framework. RESULTS: We identified 77 eligible systematic reviews. All of them included experimental studies and evaluated the effectiveness of digital education interventions in different health care disciplines or different digital education modalities. Most reviews included studies on various digital education modalities (22/77, 29%), virtual reality (19/77, 25%), and online education (10/77, 13%). Most reviews focused on health professions education in general (36/77, 47%), surgery (13/77, 17%), and nursing (11/77, 14%). The reviews mainly assessed participants' skills (51/77, 66%) and knowledge (49/77, 64%) and included data from high-income countries (53/77, 69%). Our novel conceptual framework of digital health professions education comprises 6 key domains (context, infrastructure, education, learners, research, and quality improvement) and 16 subdomains. Finally, we identified 61 unique questions for future research in these reviews; these mapped to framework domains of education (29/61, 47% recommendations), context (17/61, 28% recommendations), infrastructure (9/61, 15% recommendations), learners (3/61, 5% recommendations), and research (3/61, 5% recommendations). CONCLUSIONS: We identified a large number of research questions regarding digital education, which collectively reflect a diverse and comprehensive research agenda. Our conceptual framework will help educators and researchers plan, develop, and study digital education. More evidence from low- and middle-income countries is needed.

Education on Depression in Mental Health Apps: Systematic Assessment of Characteristics and Adherence to Evidence-Based Guidelines.

BACKGROUND: Suboptimal understanding of depression and mental health disorders by the general population is an important contributor to the wide treatment gap in depression. Mental health literacy encompasses knowledge and beliefs about mental disorders and supports their recognition, management, and prevention. Besides knowledge improvement, psychoeducational interventions reduce symptoms of depression, enhance help-seeking behavior, and decrease stigma. Mental health apps often offer educational content, but the trustworthiness of the included information is unclear. OBJECTIVE: The aim of this study is to systematically evaluate adherence to clinical guidelines on depression of the information offered by mental health apps available in major commercial app stores. METHODS: A systematic assessment of the educational content regarding depression in the apps available in the Apple App Store and Google Play was conducted in July 2020. A systematic search for apps published or updated since January 2019 was performed using 42matters. Apps meeting the inclusion criteria were downloaded and assessed using two smartphones: an iPhone 7 (iOS version 14.0.1) and a Sony XPERIA XZs (Android version 8.0.0). The 156-question assessment checklist comprised general characteristics of apps, appraisal of 38 educational topics and their adherence to evidence-based clinical guidelines, as well as technical aspects and quality assurance. The results were tabulated and reported as a narrative review, using descriptive statistics. RESULTS: The app search retrieved 2218 apps, of which 58 were included in the analysis (Android apps: n=29, 50%; iOS apps: n=29, 50%). Of the 58 included apps, 37 (64%) apps offered educational content within a more comprehensive depression or mental health management app. Moreover, 21% (12/58) of apps provided non-evidence-based information. Furthermore, 88% (51/58) of apps included up to 20 of the educational topics, the common ones being listing the symptoms of depression (52/58, 90%) and available treatments (48/58, 83%), particularly psychotherapy. Depression-associated stigma was mentioned by 38% (22/58) of the apps, whereas suicide risk was mentioned by 71% (41/58), generally as an item in a list of symptoms. Of the 58 included apps, 44 (76%) highlighted the importance of help seeking, 29 (50%) emphasized the importance of involving the user's support network. In addition, 52% (30/58) of apps referenced their content, and 17% (10/58) included advertisements. CONCLUSIONS: Information in mental health and depression apps is often brief and incomplete, with 1 in 5 apps providing non-evidence-based information. Given the unmet needs and stigma associated with the disease, it is imperative that apps seize the opportunity to offer quality, evidence-based education or point the users to relevant resources. A multistakeholder consensus on a more stringent development and publication process for mental health apps is essential.

Digital Behavior Change Interventions for the Prevention and Management of Type 2 Diabetes: Systematic Market Analysis.

BACKGROUND: Advancements in technology offer new opportunities for the prevention and management of type 2 diabetes. Venture capital companies have been investing in digital diabetes companies that offer digital behavior change interventions (DBCIs). However, little is known about the scientific evidence underpinning such interventions or the degree to which these interventions leverage novel technology-driven automated developments such as conversational agents (CAs) or just-in-time adaptive intervention (JITAI) approaches. OBJECTIVE: Our objectives were to identify the top-funded companies offering DBCIs for type 2 diabetes management and prevention, review the level of scientific evidence underpinning the DBCIs, identify which DBCIs are recognized as evidence-based programs by quality assurance authorities, and examine the degree to which these DBCIs include novel automated approaches such as CAs and JITAI mechanisms. METHODS: A systematic search was conducted using 2 venture capital databases (Crunchbase Pro and Pitchbook) to identify the top-funded companies offering interventions for type 2 diabetes prevention and management. Scientific publications relating to the identified DBCIs were identified via PubMed, Google Scholar, and the DBCIs' websites, and data regarding intervention effectiveness were extracted. The Diabetes Prevention Recognition Program (DPRP) of the Center for Disease Control and Prevention in the United States was used to identify the recognition status. The DBCIs' publications, websites, and mobile apps were reviewed with regard to the intervention characteristics. RESULTS: The 16 top-funded companies offering DBCIs for type 2 diabetes received a total funding of US $2.4 billion as of June 15, 2021. Only 4 out of the 50 identified publications associated with these DBCIs were fully powered randomized controlled trials (RCTs). Further, 1 of those 4 RCTs showed a significant difference in glycated hemoglobin A1c (HbA1c) outcomes between the intervention and control groups. However, all the studies reported HbA1c improvements ranging from 0.2% to 1.9% over the course of 12 months. In addition, 6 interventions were fully recognized by the DPRP to deliver evidence-based programs, and 2 interventions had a pending recognition status. Health professionals were included in the majority of DBCIs (13/16, 81%,), whereas only 10% (1/10) of accessible apps involved a CA as part of the intervention delivery. Self-reports represented most of the data sources (74/119, 62%) that could be used to tailor JITAIs. CONCLUSIONS: Our findings suggest that the level of funding received by companies offering DBCIs for type 2 diabetes prevention and management does not coincide with the level of evidence on the intervention effectiveness. There is considerable variation in the level of evidence underpinning the different DBCIs and an overall need for more rigorous effectiveness trials and transparent reporting by quality assurance authorities. Currently, very few DBCIs use automated approaches such as CAs and JITAIs, limiting the scalability and reach of these solutions.

Conversational Agent for Healthy Lifestyle Behavior Change: Web-Based Feasibility Study.

BACKGROUND: The rising incidence of chronic diseases is a growing concern, especially in Singapore, which is one of the high-income countries with the highest prevalence of diabetes. Interventions that promote healthy lifestyle behavior changes have been proven to be effective in reducing the progression of prediabetes to diabetes, but their in-person delivery may not be feasible on a large scale. Novel technologies such as conversational agents are a potential alternative for delivering behavioral interventions that promote healthy lifestyle behavior changes to the public. OBJECTIVE: The aim of this study is to assess the feasibility and acceptability of using a conversational agent promoting healthy lifestyle behavior changes in the general population in Singapore. METHODS: We performed a web-based, single-arm feasibility study. The participants were recruited through Facebook over 4 weeks. The Facebook Messenger conversational agent was used to deliver the intervention. The conversations focused on diet, exercise, sleep, and stress and aimed to promote healthy lifestyle behavior changes and improve the participants' knowledge of diabetes. Messages were sent to the participants four times a week (once for each of the 4 topics of focus) for 4 weeks. We assessed the feasibility of recruitment, defined as at least 75% (150/200) of our target sample of 200 participants in 4 weeks, as well as retention, defined as 33% (66/200) of the recruited sample completing the study. We also assessed the participants' satisfaction with, and usability of, the conversational agent. In addition, we performed baseline and follow-up assessments of quality of life, diabetes knowledge and risk perception, diet, exercise, sleep, and stress. RESULTS: We recruited 37.5% (75/200) of the target sample size in 1 month. Of the 75 eligible participants, 60 (80%) provided digital informed consent and completed baseline assessments. Of these 60 participants, 56 (93%) followed the study through till completion. Retention was high at 93% (56/60), along with engagement, denoted by 50% (30/60) of the participants communicating with the conversational agent at each interaction. Acceptability, usability, and satisfaction were generally high. Preliminary efficacy of the intervention showed no definitive improvements in health-related behavior. CONCLUSIONS: The delivery of a conversational agent for healthy lifestyle behavior change through Facebook Messenger was feasible and acceptable. We were unable to recruit our planned sample solely using the free options in Facebook. However, participant retention and conversational agent engagement rates were high. Our findings provide important insights to inform the design of a future randomized controlled trial.